Skip to content

[Exporter.Prometheus] Improve performance#7279

Merged
martincostello merged 100 commits into
open-telemetry:mainfrom
martincostello:improve-prometheus-performance
Jun 5, 2026
Merged

[Exporter.Prometheus] Improve performance#7279
martincostello merged 100 commits into
open-telemetry:mainfrom
martincostello:improve-prometheus-performance

Conversation

@martincostello
Copy link
Copy Markdown
Member

@martincostello martincostello commented May 9, 2026

Changes

Improve the performance of PrometheusSerializer by:

  • Using SearchValues<T> on .NET 8+
  • Writing common label value types directly instead of relying on ToString()
  • Formatting numeric values directly as UTF-8 on .NET 8+
  • Caching serialized metric names, metadata names, units, and static tag prefixes
  • Reusing serialized tags across histogram bucket/sum/count output

See #7279 (comment) for results.

Supersedes #7170.

Merge requirement checklist

  • CONTRIBUTING guidelines followed (license requirements, nullable enabled, static analysis, etc.)
  • Unit tests added/updated
  • Appropriate CHANGELOG.md files updated for non-trivial changes
  • Changes in public API reviewed (if applicable)

- Add support for configuring OpenTelemetry.Exporter.Prometheus.HttpListener with the `OTEL_EXPORTER_PROMETHEUS_HOST` and `OTEL_EXPORTER_PROMETHEUS_PORT` environment variables.
- Remove field for UriPrefixes and use auto-property.
- Remove `UriPrefixes` from the README.

Fixes open-telemetry#4158.
Fixes open-telemetry#7154.
Add coverage for invalid environment variables.
Remove duplicated constructor declaration.
- Add missing SHOULD requirement to specify the `escaping` value for 1.0.0 protocols.
- Update `PrometheusSerializer` to be compliant with `escaping=underscores` when using OpenMetrics.
Fix missing prefixing for metrics that start with a digit.
- Use canonical representations for numbers for "le" label values of histograms and "quantile" label values of summary metrics for OpenMetrics.
- Resolve TODO by moving check outside loop.

See https://prometheus.io/docs/specs/om/open_metrics_spec/#considerations-canonical-numbers.
Remove branches that could not be reached.
- Check destination size.
- Update CHANGELOGs.
Fix incorrect serialized value for `PrometheusType.Untyped` when using OpenMetrics.
Omit histogram `_sum` and `_count` in OpenMetrics when negative bucket thresholds are present.
Export `{name}_created` series for counters and histograms when start time is available when using OpenMetrics.
Add missing `TYPE` metadata.
- Remove non-spec `TYPE` for `_counter`.
- Fix-up timestamp precision.
Fix-up duplicated definitions.
Add missing suffix.
- Emit OpenMetrics scope metadata as a single `otel_scope` metric family with `otel_scope_info` samples instead of repeating metadata for every scope.
- Include instrumentation scope metadata on samples using `otel_scope_*` labels, including scope version, schema URL, and prefixed scope attributes.
- Drop conflicting scope attributes named `name`, `version`, and `schema_url` to avoid collisions with generated scope labels.
Remove Go theme for .NET.
Address Copilot review feedback.
Add more test coverage for patch.
Add Prometheus text fallback `target_info` output as a gauge so resource metadata is still exposed as Info-typed metrics are unavailable for PrometheusText exposition format.
Merge colliding sanitized label keys by concatenating values in lexicographic order of the original keys.
@github-actions github-actions Bot removed the pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package label Jun 4, 2026
Revert some changes from merge with main which aren't needed.
Restore optimisations that were lost in merge.
@martincostello martincostello added pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package perf Performance related labels Jun 4, 2026
@martincostello martincostello requested a review from Copilot June 4, 2026 16:45
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR targets performance improvements in the Prometheus HttpListener exporter serialization path by reducing per-metric string conversions, optimizing escaping and numeric formatting on newer TFMs, and reusing pre-serialized data (metric names and tag sets) across repeated writes (notably histogram bucket output).

Changes:

  • Add faster escaping implementations (using SearchValues<char> on .NET 8+) and write common label value types directly to UTF-8.
  • Cache pre-serialized metric name / metadata name / unit bytes on PrometheusMetric and write them as UTF-8 bytes instead of char-by-char loops.
  • Pre-serialize histogram tag sets once per MetricPoint and reuse across bucket/sum/count output; broaden buffer-growth retry logic to include ArgumentException in addition to IndexOutOfRangeException.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusSerializerExt.cs Reuse pre-serialized tag bytes for histogram output and add helpers for writing serialized tag spans.
src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusSerializer.cs Optimize escaping and label value formatting; write cached metric/unit bytes directly to the output buffer.
src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusMetric.cs Cache ASCII byte representations of metric-related names/units; adjust OpenMetrics naming/unit handling.
src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusCollectionManager.cs Treat ArgumentException as a buffer-growth signal during serialization retries (like IndexOutOfRangeException).
Comments suppressed due to low confidence (3)

src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusCollectionManager.cs:268

  • This catch (Exception ex) when (ex is IndexOutOfRangeException or ArgumentException) introduces an unused ex variable and is less explicit than catching the two expected exception types directly. Catching IndexOutOfRangeException and ArgumentException separately keeps the intent clear and avoids the unused variable.
                    catch (Exception ex) when (ex is IndexOutOfRangeException or ArgumentException)
                    {
                        if (!IncreaseBufferSize(ref buffer))
                        {
                            throw;
                        }
                    }

src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusCollectionManager.cs:285

  • This catch (Exception ex) when (ex is IndexOutOfRangeException or ArgumentException) introduces an unused ex variable and is less explicit than catching the two expected exception types directly. Catching IndexOutOfRangeException and ArgumentException separately keeps the intent clear and avoids the unused variable.
                catch (Exception ex) when (ex is IndexOutOfRangeException or ArgumentException)
                {
                    if (!IncreaseBufferSize(ref buffer))
                    {
                        throw;
                    }
                }

src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusCollectionManager.cs:337

  • This catch (Exception ex) when (ex is IndexOutOfRangeException or ArgumentException) introduces an unused ex variable and is less explicit than catching the two expected exception types directly. Catching IndexOutOfRangeException and ArgumentException separately keeps the intent clear and avoids the unused variable.
                catch (Exception ex) when (ex is IndexOutOfRangeException or ArgumentException)
                {
                    if (!IncreaseBufferSize(ref buffer))
                    {
                        throw;
                    }
                }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Revert change from merge that is now redundant (and wrong).
@github-actions github-actions Bot removed the pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package label Jun 4, 2026
@martincostello
Copy link
Copy Markdown
Member Author

Copilot summary of comparing the results from running .\benchmark.ps1 @("*Prometheus*") -Runtimes @("net10.0") -EnableMemoryDiagnoser:

improve-prometheus-performance outperformed main on every benchmark. Ratios below are improve-prometheus-performance / main, so values below 1.0x indicate improvements.

Benchmark Parameter Duration (main -> branch) Duration ratio Allocations (main -> branch) Allocation ratio
ScrapeEndpoint Accept=application/openmetrics-text; version=1.0.0; charset=utf-8 176.3 us -> 113.5 us 0.644x 363.98 KB -> 150.95 KB 0.415x
ScrapeEndpoint Accept=text/plain 150.5 us -> 100.7 us 0.669x 330.67 KB -> 117.63 KB 0.356x
WriteHistogramMetric NumberOfSerializeCalls=1 7.19 us -> 1.51 us 0.210x 20.39 KB -> 1.41 KB 0.069x
WriteHistogramMetric NumberOfSerializeCalls=1000 7.09 ms -> 1.46 ms 0.206x 20390.63 KB -> 1414.06 KB 0.069x
WriteHistogramMetric NumberOfSerializeCalls=10000 71.43 ms -> 15.42 ms 0.216x 203906.25 KB -> 14140.63 KB 0.069x
WriteMetric NumberOfSerializeCalls=1 8.96 us -> 2.88 us 0.321x 23.82 KB -> 4.74 KB 0.199x
WriteMetric NumberOfSerializeCalls=1000 8.98 ms -> 4.91 ms 0.547x 23820.31 KB -> 4742.19 KB 0.199x
WriteMetric NumberOfSerializeCalls=10000 90.00 ms -> 27.73 ms 0.308x 238203.13 KB -> 47421.88 KB 0.199x
WriteMetricWithTypedLabels NumberOfSerializeCalls=1 628.0 ns -> 495.9 ns 0.790x 1.41 KB -> 1.31 KB 0.929x
WriteMetricWithTypedLabels NumberOfSerializeCalls=1000 621.6 us -> 490.3 us 0.789x 1414.06 KB -> 1312.50 KB 0.928x
WriteMetricWithTypedLabels NumberOfSerializeCalls=10000 6.29 ms -> 4.77 ms 0.759x 14140.63 KB -> 13125.00 KB 0.928x

Largest improvements were in WriteHistogramMetric, where duration dropped to about 0.206x-0.216x of baseline and allocations dropped to 0.069x. ScrapeEndpoint also improved materially, with duration at 0.644x-0.669x and allocations at 0.356x-0.415x of baseline.

@martincostello martincostello marked this pull request as ready for review June 5, 2026 06:43
@martincostello martincostello requested a review from a team as a code owner June 5, 2026 06:43
Copy link
Copy Markdown
Member

@Kielek Kielek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One NIT + some codex feedback

[P2] src/OpenTelemetry.Exporter.Prometheus.HttpListener/Internal/PrometheusSerializer.cs:257: the new typed label-value formatter is only used on the fast path. When WriteTags falls back for sanitized key collisions, AddLabel still stores values through GetLabelValueString at line 956, which uses current-culture ToString() for everything except bool/double/float at line 570. That makes the same label value serialize differently depending on whether a collision occurs. For example, under fr-FR, a non-colliding decimal point tag now emits 1.23, but colliding tags that go through merge can emit 1,23;.... The fallback path should use the same invariant formatting logic as WriteLabelValue(object?).

- Add comment.
- Fix slow-path to also use invariant formatting.
- Simplify some tests.
@martincostello martincostello added this pull request to the merge queue Jun 5, 2026
Merged via the queue into open-telemetry:main with commit 7c6622e Jun 5, 2026
75 checks passed
@martincostello martincostello deleted the improve-prometheus-performance branch June 5, 2026 09:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

keep-open Prevents issues and pull requests being closed as stale perf Performance related pkg:OpenTelemetry.Exporter.Prometheus.HttpListener Issues related to OpenTelemetry.Exporter.Prometheus.HttpListener NuGet package

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants